Distortion Correction Device for Correcting Imaged Object to Produce Plane 

Image without Distortion 

[000 1] This appHcation is based on application No. 2000-260360 filed in 
5 Japan, the content of which is hereby incorporated by reference. 
BACKGROUND OF THE INVENTION 
Field of the Invention 

[0002] The present invention relates to a device and a method for 
correcting distortion and a computer -readable recording medium recorded 

10 thereon a distortion correction program. In particular, the invention 

relates to a distortion correction device and a distortion correction method 
for imaging an original document occupying two facing pages of an opened 
book or the hke to reproduce the document as a plane image thereof, and 
relates to a computer -readable recording medium recorded thereon a 

15 program for operating a computer as a distortion correction device. 
D escrip tion of the Related Art 

[0003] When an original document on two facing pages of a bound book 
or the Hke is imaged from above, a resultant image is distorted due to the 
nonuniform height from a plate on which the document is placed 

20 (hereinafter referred to as "platen") to the surface (to be imaged) of the 

document. Figs. 2 1, 2 lA and 2 IB show an exemplary distorted image of an 
original document on two facing pages of an opened book (the document in 
this state is hereinafter referred to as "double page spread document") that 
is imaged from above. 

25 [0004] Referring to Figs. 21, 21A and 2 IB, the image produced by 

imaging a double page spread document 2 has distortion of character hnes 
as found in a circled region a (Fig. 21 A) where characters are rotated 
differently depending on positions on the image. Moreover, character size 
differs depending on positions on the image as shown in a circled region h 

30 (Fig. 2 IB) where characters are further deformed. It is thus difficult to 

recognize characters from the distorted image. This problem of distortion is 
found not only in the character region but also in any photo regions and any 
regions enclosed by ruled hnes. 
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[0005] The image distortion is briefly described below in conjunction 
with Figs, 22A, 22B and 23 separately for distortions in the vertical and 
horizontal directions of an image of an original document. 
[0006] Distortion occurring in the vertical direction is described first 
5 with reference to Fig, 22A which schematically shows double page spread 
document 2 viewed from one side. An imaging unit is positioned at point 
Pi above nearly the center (point PC) of double page spread document 2. 
Points A and B are at different heights respectively from the platen and 
accordingly apart from point PI of the imaging unit by different vertical 

10 distances respectively. 

[0007] Fig. 22B shows an image of the document having its portions of 
different sizes depending on horizontal positions on the image. 
[0008] Different vertical distances from the imaging unit cause image 
size to vary. Specifically, as shown by solid Hnes in Fig. 22B, an image 

15 portion at a shorter vertical distance has a greater size while an image 

portion at a longer vertical distance has a smaller size. Then, depending on 
the position in the horizontal direction of the document, i.e., depending on 
the height from the platen, image portions have different sizes and thus the 
image portions differ from each other in terms of resolution. 

20 [0009] Horizontal distortion is described next with reference to Fig. 23 
which shows expansion and contraction of an original document occurring in 
the horizontal direction of the document. The document is imaged under 
the same condition as that shown in Figs. 22A and 22B, and Fig 23 also 
schematically shows double page spread document 2 viewed from one side. 

25 The imaged surface of double page spread document 2 is curved. 

Accordingly, portions on double page spread document 2 at even intervals 
are converted into image portions at uneven intervals on a plane as shown 
by Figs. 24 A and 24B. The resultant image thus has expanded and 
contracted portions in the horizontal direction. 

30 [00 10] A document which should originally be imaged as a plane image 
(Fig. 24A) is thus imaged as an image distorted in the horizontal and 
vertical directions (Fig. 24B). 

[00 1 1] Various methods have been proposed for correcting such image 
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distortion (hereinafter referred to as "book distortion"). For example, U.S. 
Patent Nos. 5,362,958 and 5,652,805 disclose techniques of correcting book 
distortion by using spot Kght and slit light respectively to detect a surface 
shape of an original document. Japanese Patent Laying-Open No. 9- 
5 102854 discloses a technique of correcting book distortion by using a mirror 
which is set beside an original document to detect variation of height of the 
document viewed from the side thereof. Further, Japanese Patent Laying- 
Open No. 2000-20682 discloses a technique of making corrections by 
calculating variation in the height direction of an original document 
10 according to the state of distortion of an end of the document. 

[00 12] The correction methods as described above have following 
problems. 

[00 13] Specifically, the technique disclosed in U.S. Patent No. 5,362,958 
requires measurements of numerous positions for enhancing precision in 

15 detection of the surface shape of the document, namely precision in detection 
of height. Moreover, a sufficient amount of spot light must be secured for 
taldng accurate measurements. A resultant problem is thus an increased 
device size and a longer processing time. In addition, a correction process is 
required for removing the influence of the spot light from the image in order 

20 to produce a final document image. Similarly, the technique disclosed in 
U.S. Patent No. 5,652,805 has a problem in terms of the amount of slit light 
and requirement of correction process for eliminating the influence of the 
slit light. 

[00 14] The technique shown in Japanese Patent Laying-Open No, 9- 
25 102854 has a problem that an imaging unit and an object to be imaged are 

limited in position because the set mirror is used for detecting the height. 

[00 15] The technique shown in Japanese Patent Laying-Open No. 

2000-20682 is intended to be apphed to a contact type scanner for a fixed 

original image and a fixed imaging unit and thus is difficult to be applied to 
30 a hand-held type imaging unit or the like for an original document to be 

imaged and an imaging unit at respective relative positions that are not 

constant. 

[0016] Namely, a problem resulting from the inconstant relative 
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positions is that it is impossible to distinguish an end shape of a document 
having a remarkably varying height, the document viewed from almost 
directly above, from an end shape of a document having a moderately 
changing height, the document viewed in an angled direction. Then, 
5 variation in the height of the document cannot be specified resulting in an 
inaccurate correction. 
SUMMARY OF THE INVENTION 

[0017] One object of the present invention is accordingly to provide a 
distortion correction device, a distortion correction method and a computer- 

10 readable recording medium recorded thereon a distortion correction 

program for easily and accurately correcting distortion of an image produced 
by imaging a surface of an object, the distortion caused by nonuniform 
height of the surface, regardless of imaging conditions. 
[00 18] A distortion correction device according to one aspect of the 

15 present invention includes: an imaging unit for imaging an object from a 

point in the direction of height of a surface of the object to generate an image 
signal representing an image of the surface of the object; an image signal 
acquiring unit for acquiring the image signal generated by the imaging unit; 
a detecting unit for detecting a line image from the acquired image signal; a 

20 relative position calculating unit for calculating a relative position of the 

imaging unit with respect to the object according to the line image detected 
by the detecting unit; a height distribution calculating unit for calculating a 
height distribution of the surface of the imaged object in the direction of 
height according to the calcvdated relative position and the detected line 

25 image; and a processing unit for processing the image signal to correct 
distortion of the image of the object according to the calculated height 
distribution. 

[00 19] According to the detected Hne image and information about the 
relative position of the imaging unit with respect to the object, the height 
30 distribution of the surface of the imaged object is calculated. Accordingly, 
when the position of the imaging unit changes, for example, when a double 
page spread document having different heights from a plate on which the 
document is placed is imaged from above by a hand-held type imaging unit, 
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information about a relative position of the imaging unit with respect of the 
object (document) is acquired to enable accurate calculation of the height 
distribution to be made according to the acquired information. 
[0020] On the basis of information about the calculated height 
5 distribution, distortion of the image represented by the image signal is thus 
corrected. 

[002 1] In this way, it is possible to provide the distortion correction 
device capable of easily and accurately correcting any distortion of an image 
produced by imaging a surface of an object, the distortion caused by 
10 nonuniform height of the surface of the object, regardless of imaging 
conditions. 

[0022] Preferably, the line image includes an image of an end of the 
object, an image of a continuous Hne or a character string formed on the 
surface of the object. 

15 [0023] The detecting unit detects the image of an end of the object, 

image of a continuous Hne or character string formed on the surface of the 
object and thus image data about appropriate hues on the entire object is 
obtained. Ruled lines or the like can be detected to obtain line data even if 
an end of the object, for example, an end of a document, is not included in 

20 the produced image. 

[0024] Preferably, the relative position calculating unit calculates the 
relative position as a distance in the direction of height between the imaging 
unit and the surface of the imaged object and a distance in a lateral direction 
crossing the direction of height between the imaging unit and the surface of 

25 the imaged object. 

[0025] Information about the relative position of the imaging unit with 
respect to the object includes information about the distance in the direction 
of height of the object between the imaging unit and the object and 
information about the distance in the lateral direction therebetween. The 

30 positional relation between the imaging unit and the object is thus clarified 
to maiie it possible to easily and accurately calculate the height distribution 
of the surface of the imaged object. 

[0026] The information about the distance in the direction of height 

- 5 - 



means a length of a perpendicular extended from the imaging unit to the 
imaged object or information about an approximate distance. The 
information about the distance in the lateral direction means information 
about a position of a foot of a perpendicular extended from the imaging unit 
5 to the imaged object, for example. 

[0027] Preferably, the detecting unit detects two line images from the 
acquired image signal, and the relative position calculating unit calculates 
the distance in the lateral direction between the imaging unit and the 
surface of the imaged object according to inclination of the two line images. 

10 [0028] The distance in the lateral direction between the imaging unit 

and the object is calculated according to incHnation of the detected two line- 
shaped regions. As local inclination of the Une-shaped regions is used, the 
lateral distance can be calculated through a simple process. 
[0029] Still preferably, the distortion correction device further includes 

15 an edge detecting unit for detecting an edge of the image and an edge image 
generating unit for generating an edge image with the detected edge 
enhanced. The detecting unit refers to the edge image to detect the line 
image. 

[0030] The Hne-shaped regions are detected according to the edge- 
20 enhanced image and thus the detecting process can simply and surely be 
performed, 

[003 1] StUl preferably, the image signal acquiring unit includes a pre- 
processing unit for performing a predetermined pre-processing on the 
acquired image signal, and the pre-processing unit performs at least one of 
25 scaling up/down, sharpness enhancement, white pixel expansion and 
smoothing. 

[0032] The edge-enhanced image is produced on the basis of the image 
data which has undergone a predetermined pre-processing such as scaHng 
up/down. This pre-processing facilitates production of the edge-enhanced 
30 image and accordingly the image can be produced more appropriately. 

[0033] According to another aspect of the invention, a camera includes: 
an imaging unit for converting an optical image of an object into an electric 
signal to generate an image signal; and a processor for calculating a 
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positional relation between the imaging unit and the object according to the 
image signal and correcting the image signal according to the calculated 
positional relation to represent an image of the object as a plane object. 
[00341 The camera generates the image signal by converting the optical 
image of the object into the electric signal, calculates the positional relation 
between the imaging unit and the object according to the image signal, and 
corrects the image signal according to the calculated positional relation to 
represent an image of the object as a plane object. The camera can thus be 
provided that can easily and accurately make correction to represent an 
image of the object as a plane object regardless of imaging conditions. 
[0035] According to still another aspect of the invention, a camera 
includes: an imaging unit for converting an optical image of an object into an 
electric signal to generate an image signal; a first calculating unit for 
calculating respective heights of parts of the object according to the image 
signal; a second calculating unit for calculating a positional relation 
between the object and the imaging unit according to the image signal; and a 
correcting unit for correcting an image of the object represented by the 
image signal to scale up or down the image according to the respective 
heights of the parts of the object calculated by the first calculating unit and 
the positional relation calculated by the second calculating unit. 
[0036] According to a farther aspect of the invention, a method of 
correcting an image produced by imaging a surface of an opened book facing 
upward from above of the surface of the book by a camera having a non -fixed 
positional relation with the book, includes the steps of: imaging the surface 
of the opened book to generate an electric signal representing the image of 
the surface of the book; measuring a distance between the camera and the 
surface of the book; extracting, according to the electric signal, an edge 
image corresponding to an upper or lower end of the book in the image 
represented by the electric signal; determining a position of the camera 
relative to the surface of the book according to the extracted edge image; 
determining a height distribution of the upper or lower end of the surface of 
the book according to the measured distance, the extracted edge image of the 
upper or lower end of the book, and the position of the camera relative to the 
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surface of the book; determining a height distribution of an entire image of 
the surface of the book on the supposition that the book is at the same height 
in the direction in which the book is opened; and converting, according to the 
determined height distributions, the image signal into a signal representing 
5 an image of the surface of the book as a plane surface. 

[0037] According to a further aspect of the invention, a program is 
provided for executing a method of correcting an image produced by imaging 
a surface of an opened book facing upward from above of the surface of the 
book by a camera having a non-fixed positional relation with the book, the 

10 method being executed by a computer included in the camera. The method 
includes the steps of: imaging the surface of the opened book to generate an 
electric signal representing the image of the surface of the book; measuring 
a distance between the camera and the surface of the book; extracting, 
according to the electric signal, an edge image corresponding to an upper or 

15 lower end of the book in the image represented by the electric signal; 

determining a position of the camera relative to the surface of the book 
according to the extracted edge image; determining a height distribution of 
the upper or lower end of the surface of the book according to the measured 
distance, the extracted edge image of the upper or lower end of the book, and 

20 the position of the camera relative to the surface of the book; determining a 
height distribution of an entire image of the surface of the book on the 
supposition that the book is at the same height in the direction in which the 
book is opened; and converting, according to the determined height 
distributions, the image signal into a signal representing an image of the 

25 surface of the book as a plane surface. 

[0038] According to a further aspect of the invention, a program product 
is used for executing, by a computer, a method of correcting an image 
produced by imaging a surface of an opened book facing upward from above 
of the surface of the book by a camera having a non-fixed positional relation 

30 with the book, the computer provided separately from the camera. The 
method includes the steps of: reading an electric signal representing the 
image of the surface of the book, the electric signal generated by imaging the 
surface of the opened book; inputting a distance between the camera, and the 

-8- 



surface of the book; extracting, according to the electric signal, an edge 
image corresponding to an upper or lower end of the book in the image 
represented by the electric signal; determining a position of the camera 
relative to the surface of the book according to the extracted edge image; 
5 determining a height distribution of the upper or lower end of the surface of 
the book according to the measured distance, the extracted edge image of the 
upper or lower end of the book, and the position of the camera relative to the 
surface of the book; determining a height distribution of an entire image of 
the surface of the book on the supposition that the book is at the same height 

10 in the direction in which the book is opened; and converting, according to the 
determined height distributions, the image signal into a signal representing 
an image of the surface of the book as a plane surface, 
[0039] The programs each may be recorded on a recording medium. 
[0040] According to the present invention, a distortion correction 

15 program or a computer-readable recording medium on which such a 

program is recorded can be provided, the program used for executing a 
distortion correction method, by a computer, for easily and accurately 
correcting distortion of an image produced by imaging a surface of an object, 
the distortion caused by nonuniform height of the surface, regardless of 

20 imaging conditions. 

[0041] The foregoing and other objects, features, aspects and 
advantages of the present invention will become more apparent from the 
following detailed description of the present invention when taken in 
conjunction with the accompanying drawings. 

25 BRIEF DESCRIPTION OF THE DRAWINGS 

[0042] Fig. 1 schematically shows a hand-held imaging unit including a 
distortion correction device that images a double page spread document 
according to one embodiment of the present invention. 
[0043] Fig. 2 is a block diagram schematically showing an entire 

30 structure of the distortion correction device according to the embodiment of 
the present invention. 

[0044] Fig. 3 is a flowchart showing an entire flow of correcting book 
distortion by the distortion correction device. 
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[0045] Fig. 4 shows an image of a double page spread document shown 
in Fig. 21 that has been scaled down and thereafter subjected to sharpness 
enhancement, white pixel expansion and smoothing successively. 
[0046] Fig. 5 shows a secondary differential image produced by 
calculating absolute values of secondary derivatives on the basis of the pre- 
processed image shown in Fig. 4, 

[0047] Fig. 6 illustrates detection of a document end. 

[0048] Fig. 7 shows pixels in and around region A in Fig. 6 that is 

expanded for illustrating detection of edge pixels on the right, 

[0049] Figs. 8A and SB Illustrate a shape of an end of a double page 

spread document of a thin book viewed by an imaging unit at a considerably 

sharp angle. 

[0050] Figs. 9A and 9B illustrate a shape 1' of an end of a document of a 
thick book viewed from almost directly above. 

[0051] Figs. lOA and lOB illustrate a shape of an end of a large-size 
document of a thin book viewed from almost directly above. 
[0052] Fig. 11 illustrates a specific process of calculating a relative 
position from information about document ends. 

[0053] Fig. 12 illustrates calcxilation of a relative position of an imaging 

unit according to a plurahty of angles of inclination. 

[0054] Fig. 13 is a perspective view showing a positional relation 

between an end L of an imaged document and an imaging unit 1. 

[0055] Fig. 14 illustrates the positional relation viewed in the direction 

of the arrow shown in Fig, 13. 

[0056] Figs, 15A and 15B conceptually show geometric deformation in 
the horizontal direction. 

[0057] Fig. 16 illustrates conversion of an image through the geometric 
deformation in detail. 

[0058] Figs. 17A to 17C conceptually show correction of book distortion 
by geometric deformation correction according to height information. 
[0059] Figs. 18A and 18B illustrate a process performed when an 
entire document has skew or deflection. 

[0060] Fig. 19 illustrates a method of calculating height information 
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from a produced image of a document which has its upper and lower ends 
with different height variations due to waviness or deflection of the 
document. 

[006 1] Fig, 20 is a block diagram showing a structure of another 
embodiment of the present invention. 

[0062] Figs. 2 1, 2 lA and 2 IB show one example of a distorted image 
produced by imaging a double page spread document from above. 
[0063] Figs. 22A and 22B show variation of size in an image of a 
document depending on horizontal position in the image. 
[0064] Fig. 23 shows an image of a document having expanded and 
contracted portions in the horizontal direction. 

[0065] Figs. 24A and 24B show a plane image produced as an image 
having distortions in the horizontal and vertical directions respectively. 
DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0066] Embodiments of the present invention are hereinafter described 
in detail in conjunction with the drawings. 

[0067] Referring to Fig. 1, a distortion correction device is provided 
within a hand-held type imaging unit 1. A double page spread document 2 
is imaged from above by imaging unit 1 and the distortion correction device 
corrects any distortion of a produced image. 

[0068] Fig. 2 is a block diagram schematically showing an entire 
structure of a distortion correction device 200 according to one embodiment 
of the present invention. Referring to Fig. 2, distortion correction device 
200 includes an input unit 201 for entering image data of an original 
document imaged by imaging unit 1, a memory 203 for storing a distortion 
correction program to be executed by a CPU 209 and storing the entered 
image data, processed data and the hke, a distance sensor 205 for measuring 
the distance from imaging unit 1 to a surface of the document to be imaged, 
an output unit 207 for outputting image data with distortion corrected by 
CPU 209, and CPU 209 for controlling these units and correcting distortion. 
[0069] Image data produced by imaging the document by imaging unit 1 
is input as digital data to distortion correction device 200 from input unit 
201. The input image data is temporarily stored in a predetermined region 



in memory 203. The stored image data is appropriately read by CPU 209 
and then subjected to image processing of various types. At this time, data 
about an approximate distance from the imaging unit to the document 
surface is used as required that is measured by distance sensor 205. Final 
image data having been image-processed is suppHed to the outside of 
distortion correction device 200 from output unit 207. 
[0070] A process carried out by distortion correction device 200 is 
described below in detail. Fig. 3 is a flowchart showing an entire flow of a 
book distortion correction process by distortion correction device 200, 
Distortion correction device 200 acquires from distance sensor 205, before 
carrying out step S301, information about an approximate distance from 
hand-held imaging unit 1 to a surface of a document to be imaged. 
Distance sensor 205 may calculate this distance by any method such as 
contrast method, triangvilation, phase-contrast method and the like. If an 
original document size is known, the approximate distance may be 
calculated based on a document size in an image. 

[0071] Book distortion is then corrected. Here, for the purpose of 
presenting a simple description, it is supposed that the document is laterally 
opened on two facing pages of a book (double page spread document). It is 
possible to detect whether the document is opened in the lateral direction or 
vertical direction (perpendicxdar to the lateral direction) or rotated, 
according to a positional relation between the ends or corners of the 
document. Accordingly, correction can be made similarly on any image by 
rotating the image or coordinate axis so that the image is in the laterally 
opened state. A laterally opened document is thus described here. 
[0072] The approximate distance to the surface of the double page 
spread document is determined and thereafter image data of the document 
imaged by imaging unit 1 is pre-processed in step S301. Specifically, the 
input image data undergoes at least one of scaling down, sharpness 
enhancement, smoothing and white-pixel expansion processes and the hke. 
[0073] Fig. 4 shows an image produced by scaling down and subsequent 
sharpness enhancement, white-pixel expansion and smoothing of the image 
of the double page spread document shown in Fig. 21. This pre-processing 
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makes it possible to ensure a subsequent edge image production. 
[0074] After the pre-processing, an edge image is produced in step S303. 
Secondary differentiation, for example, is used for producing an edge image. 
Fig. 5 shows a secondary differential image as an edge image produced by 
calculating absolute values of secondary derivatives based on the pre- 
processed image shown in Fig. 4. This edge image production ensures a 
subsequent document end detection. 

[0075] After the edge image is produced, document ends are detected in 
step S305. In this step, both of the upper and lower ends of the document 
are detected because they are necessary for a subsequent detection of a 
relative position of imaging unit 1. A method of detecting document ends is 
detailed later. 

[0076] After the document end detection, in step S307, a relative 
position of imaging unit 1 with respect to the document is detected according 
to information on the detected upper and lower ends of the document. Here, 
the relative position refers to a position, on an image of the imaged 
document, of a foot of a perpendicvdar from imaging unit 1, the 
perpendicular being an imaginary line extending from imaging unit 1 that is 
perpendicular to a plane on which the document is placed. Without the 
information on the relative position and distance between the document and 
imaging unit 1, accurate information about height cannot be derived from 
shapes of the detected document ends. 

[0077] Figs. 8A to lOB show that, when ends of documents of different 
heights are detected, the detected ends have similar shapes. In these 
drawings, imaging unit 1 is positioned at point PI for detecting the upper 
end of a document and a foot of a perpendicular from imaging unit 1 is 
located at point PO. Figs. 8A, 9A and lOA each show a perspective view 
illustrating a positional relation between imaging unit 1 and a document to 
be imaged, and Figs. 8B, 9B and lOB each show a plan view of the document 
viewed from above. 

[00 78] When a double p age spread document of a thin book is viewed by 
imaging unit 1 at a considerably sharp angle, an end of the document has a 
shape 1 as shown in Figs. 8A and 8B. When a document of a thick book is 
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viewed from almost directly above, an end of the document has a shape 1' as 
shown in Figs, 9A and 9B. When a large-size document of a thin book is 
viewed from almost directly above, an end of the document has a shape 1" as 
shown in Figs. lOA and lOB. These shapes 1, 1' and 1" are similar to each 
5 other. Under this situation, difference in height cannot accurately be 

detected based on the upper ends of the documents if there is no information 
about the relative position (information about position of point PO). 
[0079] After the relative position of imaging unit 1 is detected, height is 
calculated based on the end shape of the document in step S309. 
10 Specifically, the approximate distance from imaging unit 1 to the document, 
the relative position of imaging unit 1 and information about the end shape 
of the document are used to calculate the height of each point on the 
document. 

[0080] It is assumed here that an image of a laterally opened document 
15 has less variation of height in the lengthwise direction (direction 

perpendicular to the lateral direction). Then, only the upper end of the 
document is used to calculate heights of respective points in the lateral 
direction. 

[0081] After height calculation, the document image data undergoes 
20 geometric deformation correction using the height information in a final step 
S3 11. Specifically, the image produced by imaging the document is 
geometrically converted in the horizontal direction (direction of x- axis) such 
that a curved surface is converted into a plane surface. With respect to the 
vertical direction (direction of y-axis), image portions with respective sizes 
25 expanded or contracted and thus different from each other are scaled down 
or up. 

[0082] After the geometric deformation correction, the process of 
correcting book distortion is completed. Through the process as discussed 
above, distortion of an image occurring when a double page spread 
30 document is imaged from above is corrected and thus the document can be 
reproduced which is recognized as a plane document. 
[0083] The detection of document ends (S305), detection of a relative 
position of the imaging unit (8307), height calculation (S309) and geometric 
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deformation correction (S3 11) in the process of correcting book distortion 
shown in Fig. 3 are each described in detail below. 

[0084] Detection of document ends (S305) is described in conjunction 
with Fig. 6. Referring to Fig. 6, the boundary of a rectangle represents the 
perimeter of an entire input image, curve L represents a detected end of a 
document, and broken Hnes Lv, Lvl and Lv2 are lines dividing the entire 
input image into four equal parts in the horizontal direction. 
[0085] Usually, a document image is supposed to be present surely 
about the center of an input image. Then, the uppermost continuous edge 
extending over broken lines Lvl and Lv2 is recognized as the upper end of 
the document. 

[0086] Accordingly, the uppermost edge pixel on broken line Lvl is 
detected first. Specifically, pixels on broken Line Lvl of the secondary 
differential image are examined in the order from the top pixel to determine 
whether the pixel has a value greater than a certain threshold Thl. When 
any pixel having its value greater than threshold Thl is detected, the pixel 
is recognized as the uppermost edge pixel. 

[0087] After the edge pixel on Lvl is detected, edge pixels on the right 
are detected successively. Fig. 7 is an enlarged view showing pixels in and 
around region A in Fig. 6 for illustrating detection of an edge pixel on the 
right. Referring to Fig. 7, "noted pixel P" is a pixel which has already been 
detected, and a next edge pixel among five pixels (Q1-Q5) on the right of 
noted pixel P is detected. 

[0088] First, a pixel which has the largest pixel value of the secondary 
differential image is selected from the right five pixels. Then, the pixel 
value of the selected pixel is compared with a threshold Th2 (<Thl) to 
determine whether the pixel value is greater than threshold Th2. If the 
pixel value is greater, the selected pixel is recognized as an edge pixel. 
Then, this recognized edge pixel is designated as a noted pixel to extract an 
edge pixel from the following five pixels on the right in the same manner. 
In this way, the edge is tracked successively. 

[0089] If there is no pixel with its value greater than threshold Th2 
before an edge pixel at broken line Lv2 is detected and thus the edge 
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discontinues, it is determined that the detected edge line is not a correct 
edge and then an edge extraction is newly started. A pixel on broken line 
Lvl lower than the previously extracted pixel thereon is selected as an edge 
pixel to similarly track the edge to the right. This process is appropriately 
5 repeated to extract an uppermost continuous edge extending over broken 
hnes Lvl and Lv2 as the upper end of the document, 
[0090] After the continuous edge is extracted, the edge is similarly 
tracked to the left of broken hne Lvl and to the right of broken line Lv2, 
Ultimately, the upper end of the document is detected on the secondary 

10 differential image. 

[0091] The lower end of the document is similarly detected. Namely, 
the process shown in Fig. 6 can be appUed to detection of the lower end as a 
symmetrical counteirpart of the upper end. The right and left ends are also 
detected if necessary in the similar manner to that for the upper and lower 

15 ends. 

[0092] The detected upper end of the document corresponds to the upper 
end of a scaled-down document image and thus the upper end must be 
correlated with an original image size. Then, information on document end 
image is reconstituted by any known interpolation method such as Knear 

20 interpolation and cubic convolution. 

[0093] Detection of a relative position of imaging unit 1 with respect to 
the document (S307) is described. Fig. 11 illustrates a specific process of 
calculating the relative position from the information about the document 
ends. Referring to Fig. 11, x-axis represents horizontal direction and y-axis 

25 represents vertical direction. From local incHnations of the document ends, 
the relative position is calculated. 

[0094] First, an angle of inclination 9 at the uppermost point (X, Ya) of 
imaged double page spread document 2 and an angle of inclination (j) at the 
lowermost point (X, Yb) with the same x coordinate value as that of the 
30 uppermost point are determined. If angles G and ^ have the same sign and 
relation |9| > |(j>| is satisfied, the position of imaging unit 1 in the y direction 
(relative position coordinate) Yc is determined by Yc = Yb + k x (Yb-Ya) x 
If angles 9 and ^ have the same sign and relation |9| < \^\ is satisfied, the 
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position is determined by Yc - Ya - k x (Yb - Ya) x |e|. If angles 0 and (j) have 
different signs, the position is determined by Yc = {(Ya + Yb) / 2} + {(Yb - Ya) 
/ 2} X (|(|)| - 191) / (|0| + |<i)|), 

[0095] Here, coefficient k is a uniquely set value as required depending 
on an approximate distance between imaging unit 1 and the document, a 
standard object size (or angle of view in imaging) and the like. In Fig. 11, 
the relative position of imaging unit 1 is at a y-coordinate position indicated 
by a character string S. It is noted that the angle of incHnation is positive 
in the counterclockwise direction and thus angle G is positive while angle cj) is 
negative in Fig. 11. 

[0096] Instead of local inclinations of the ends of the document used for 
calculating the relative position, inclinations of character strings, rectangles, 
hues or the like within the document may be used to similarly calculate the 
relative position. This is because the ends of a document are not 
necessarily in the produced image of the document. The incHnation angles 
are extracted by a known method, for example, the incHnation angles are 
measured after expansion of black pixels by scaling down of the image, 
expansion of black pixels, smoothing and the Hke. 

[0097] When no incHnation of a document end, character string or the 
Hke is extracted and thus the angle of incHnation is zero in the 
corresponding local region, the position of the region with zero-inclination 
may be determined as the relative position. For example, in Fig. 11, the 
region including character string S with angle of inclination of zero is 
determined as the relative position. The relative position of imaging unit 1 
may be calculated from determined incHnation angles of a pluraHty of 
regions as shown in Fig. 12 indicated by the regions enclosed by dotted Hnes 
respectively, in order to enhance the accuracy of the relative position. 
[0098] Calculation of height is described below in conjunction with Figs. 
13 and 14. Fig. 13 is a perspective view showing a positional relation 
between an imaged document end L and imaging unit 1. Fig. 14 shows a 
positional relation therebetween viewed in the direction indicated by the 
arrow in Fig. 13. 

[0099] Referring to Figs. 13 and 14, point PI represents a position of 
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imaging unit 1, point PO represents a foot of a perpendicular extending from 
PI to a plane on which the document is placed, point A' represents the 
highest position on a document end, point B' represents an arbitrary 
position on the document end, and curve L represents a shape of the imaged 
5 document end. 

[0 100] The height of point A' can be calculated by utihzing similarity of 
triangles in accordance with A A = PlPO • A2A / A2P0. Similarly, the 
height of point B' can be calculated in accordance with B'B = PlPO • B2B / 
B2P0. 

10 [0 10 1] PlPO indicates an approximate distance from imaging unit 1 to 
the document. POP is determined as a relative position of imaging unit 1 
with respect to the document. Accordingly, A2P0 and B2P0 can be 
determined from POP and respective coordinate values. 
[0 102] Geometric deformation correction (step S3 1 1) by means of the 

15 height information is described now. Figs. 15A and 15B conceptually 
illustrate geometric deformation in the horizontal direction. Fig. 15A 
shows a shape of the document surface before deformation and Fig. 15B 
shows a shape of the document surface after deformation. Referring to Figs. 
15A and 15B, the document image is geometrically deformed by dividing the 

20 document surface after deformation into equal parts at lattice points MO, Ml, 
M2 ... (M0-M9) in this order and determining respective corresponding 
positions on the surface before deformation. 

[0103] The image is thus converted through the geometric deformation 
as more detailed below in conjunction with Fig. 16. Referring to Fig. 16, 

25 image data before deformation that should be at lattice point Ml after 

deformation is the data at position Nl which satisfies the relation MOMl = 
NONl on a document end. Point Nl is determined from inclination about 
NONl (MO=NO). The inclination about NONl may be determined, from the 
height information, by calculating inclination on a curve of a position 

30 corresponding to any position near Nl. 

[0104] After point Nl is determined, position N2 is similarly determined 
that satisfies the relation M1M2 = N1N2. Data at point N2 is converted to 
lattice point M2 on the plane. Similarly, position N3 is determined that 
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satisfies the relation M2M3 = N2N3 and data at point N3 is converted to 
point M3. Tlus process is repeated to accomplish geometric deformation in 
the horizontal direction. 

[0 105] Figs. 17A to 17C conceptually illustrate correction of book 
distortion through the geometric deformation correction on the basis of the 
height information. Referring to Figs. 17A to 17C, a distorted image as 
shown in Fig. 17Ais geometrically converted in the horizontal direction into 
a horizontally expanded image as shown in Fig. 17B that has a uniform 
resolution. 

[0106] Then, the horizontally corrected image shown in Fig. 17B 
undergoes scaling up/down in the vertical direction into a final plane image 
with its vertical size variation corrected as shown in Fig. 17C. 
[0107] The scaling up/down in the vertical direction is done by means of 
the height information calculated from the shape of the document end and 
an imaging distance (approximate distance) of imaging unit 1. The image 
has different vertical sizes depending on horizontal positions as illustrated 
in Fig. 21 due to different distances firom the document to the imaging unit. 
Then, in accordance with the height information and approximate distance, 
the distance from each point on the document to the imaging unit can be 
determined and the determined distance can be used for scaling up/down in 
the vertical direction. For example, the geometric conversion is made line 
by line in the vertical direction on the basis of the highest or lowest position 
on the document for example. 

[0 108] Although Figs. 17A to 17C show that the horizontal geometric 
conversion and the vertical scaling up/down are carried out in this order, the 
conversion may be done in the inverse order. Further, in order to achieve 
fast processing, coordinate calculation only may be made in advance and 
thereafter horizontal and vertical conversions may be carried out all 
together. 

[0 109] The discussion above has been presented on the assumption that 
the imaged double page spread document has no skew or deflection. A 
process carried out when an entire image has skew or deflection is described 
below in conjunction with Figs. 18A to 19. 
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[0110] As shown in Fig. 18A, when a double page spread document 
exhibits skew or deflection, corner points A to D are determined according to 
results of the document end detection (S305) in Fig. 3. The comer points 
are corrected in advance as shown in Fig. 18B. Subsequent steps S307, 
S309 and S3 11 in Fig. 3 are carried out accordingly. 

[0111] If a p ositional relation in terms of sp ace between the double p age 
spread document and imaging unit 1 is known, the positional relation may 
be used to correct skew or deflection. In this case, image data with 
deflection or the Hke which has already been corrected may be used to 
correct book distortion thereof shown in Fig. 3 (steps S301-S311). 
[0112] Fig. 19 illustrates a method of calculating height from an image 
produced by imaging a document which has its upper and lower ends with 
different height variations due to waviness or deflection of the document. 
Here, straight hnes AM, ST and DN are (horizontal) lines in parallel with 
X-axis. Straight hne QR corresponds to a vertical line on the document. 
Line QR is not in parallel with y-axis on the image of the document due to 
deflection. 

[0113] Referring to Fig. 19, height of an arbitrary point on hne AM 
(y=ya) is calculated from the shape of the upper end of the document by the 
above-described method. Similarly, height of an arbitrary point on Hne DN 
(y=yd) is calculated from the shape of the lower end of the document. Then, 
height information Hq of point Q on Hne AM and height information Hr of 
point R on Hne DN can be calculated if respective x-coordinate values xq and 
xr are known: 

[0114] xq = {(xm - xa) (xp - xs) / (xt - xs)} + xa 
[0115] xr = {(xn - xd) (xp - xs) / (xt - xs)} + xd 

where xi (i = a, d, m, n, p, t) is a value of an x coordinate at point i. Xq and 

xr can thus be used to calculate height information Hp of an arbitrary point 

P on Hne ST by means of the foUowing equation: 

[0116] Hp = {(yd - yp) Hq + (yp - ya) Hr} / (yd - ya). 

[0117] Height of an arbitrary point on the document is accordingly 

calculated by using respective distances from the upper and lower ends of 

the document and height information of the upper and lower ends of the 
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document. 

[0118] Geometric conversion on the basis of the height information is 
carried out in the horizontal direction by coordinate position transformation 
based on height distribution on a horizontal Hne (line ST) passing an 
5 arbitrary point P and in the vertical direction by coordinate position 

transformation based on height information of arbitrary point P (or based on 
an average of heights of aU points on QR). 

[0119] In this way, information about two heights are used to correct 
book distortion and thus distortion can be corrected with a higher precision. 

10 [0 120] Another embodiment of the present invention is now described. 
Referring to Fig. 20, the embodiment can be implemented by executing a 
distortion correction program by a computer. An image taken by a digital 
camera 310 is supphed to a computer 300 to correct any distortion of the 
image by computer 300. Computer 300 includes an interface 302 for input 

15 of image data from digital camera 310 via a cable 304, a reading unit 307 for 
input of a distortion correction program via a recording medium 306, a 
reading unit 303 for reading of recording medium 306 on which the program 
is recorded, and a processing unit 301 for controUing the entire computer. 
[0121] The distortion correction program is input to computer 300 via 

20 recording medium 306 and computer 300 uses the program to correct 
distortion of image data from digital camera 310. 

[0 122] The distortion correction program may preliminarily be installed 
on a hard disk within the computer or recorded on a removable recording 
medium such as CD-ROM and magnetic tape. In any case, the distortion 

25 correction program is recorded on a computer-readable recording medium, 
[0 123] The computer-readable recording medium may be any medium 
which fixedly holds a program, such as tape media like magnetic tape and 
cassette tape, disk media like magnetic disk (flexible disk, hard disk and the 
hke) and optical disk (CD-ROM, MO, MD, DVD and the like), card media 

30 Kke IC card (including memory card) and optical card, and semiconductor 
memories Uke mask ROM, EPROM, EEPROM and flash ROM, 
[0124] As heretofore discussed, distortion correction device 200 
according to the embodiment of the present invention detects ends of a 
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double page spread document with different heights that is imaged from 
above, the ends being detected from the resultant image of the document. 
Then, the detected document ends, an approximate distance between an 
imaging unit and the document (vertical distance) and a relative distance 
5 (horizontal distance) are used to calculate height of the document. The 
calculated height is used to geometrically convert the original image. In 
this way, the original image is converted into an image as if the double page 
spread document which actually has a distorted surface is a plane document. 
Consequently, the document can easily be recognized and image processing 

10 of the document is facilitated, 

[01251 Moreover, the shape of a document end (edge shape) detected 
from a document image can be used as a basis for determining a relative 
distance, height information and the hke. Need for illumination device, 
complex sensor and the hke is thus ehminated and consequently the 

15 correction device can be reduced in complexity and size. 

[0 126] The medium may temporarily hold the program to allow the 
program to be downloaded from any network. 

[0127] Contents stored on the recording medium are not limited to the 
program and the contents may be data. 

20 [0 128] According to the embodiment above, a document end is detected 
to determine the shape of the document end (edge shape) as described in 
conjunction with Fig. 6 for example. Not only the document end but also 
hne-shaped region on the document such as character string and ruled line 
may be detected as an edge. Depending on variation of height and 

25 direction of the document, the hne region or the hke also exhibits distortion 
on the produced image of the document as the document end does. 
Therefore, based on the shape of the hne region or the hke, height can be 
calculated. The detection of such a shape may be done by a similar method 
to that for detecting inchnation of character string, rectangle, hne and the 

30 hke used for calculating a relative position as discussed above. 

[0129] Regarding document end detection (S305) shown in Fig, 3, 
description above is apphed to detection of a document end by the method as 
shown in Figs. 6 and 7. The method is not hmited to that as described 
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above and another method may be used to detect a document end by 
increasing/decreasing the number of pixels to be compared for tracking an 
edge or still another method may be used. 

[0130] Although the present invention has been described and 
illustrated in detail, it is clearly understood that the same is by way of 
illustration and example only and is not to be taken by way of limitation, the 
spirit and scope of the present invention being limited only by the terms of 
the appended claims. 
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